Robust Lip Tracking by Combining Shape, Color and Motion

Accurately tracking facial features requires coping with the large variation in appearance across subjects and the combination of rigid and non-rigid motion. In this paper, we describe our work toward developing a robust method of tracking facial features, in particular, lip contours, by using a multi-state mouth model and combining lip color, shape and motion information. Three lip states are explicitly modeled: open, relatively closed, and tightly closed. The gross shapes of lip contours are modeled by using different lip templates. Given the initial location of the lip template in the first frame, the lip and skin color is modeled by a Gaussian mixture. Several points of a lip are tracked over the image sequence, and the lip contours are obtained by calculating the corresponding lip template parameters. The color and shape information is used to obtain lip states. Our method has been tested on 5000 images from the University of Pittsburgh-Carnegie Mellon University (Pitt-CMU) Facial Expression AU Coded Database, which includes image sequences of children and adults of European, African, and Asian ancestry. The subjects were videotaped while displaying a wide variety of facial expressions with and without head motion. Accurate tracking results were obtained in 99% of the image sequences. Processing speed on a Pentium II 400MHZ PC was approximately 4 frames/second. The multi-state model based method accurately tracked lip motion and was robust to variation in facial appearance among subjects, specularity, mouth state, and head motion.